Display apparatus, image processing apparatus, and control method

ABSTRACT

A display apparatus includes a display screen, and a controller that causes the display screen to display a composite image in which a first image acquired by imaging a space by a camera and a second image representing at least one type of aerosol existing in the space are combined. The position of the at least one type of aerosol as seen in a depth direction in the first image is reflected in the second image.

BACKGROUND 1. Technical Field

The present disclosure relates to a display apparatus, an imageprocessing apparatus, and a control method.

2. Description of the Related Art

A terminal apparatus is known that visually displays a substance thatfloats in the air, such as pollen or dust, in the form of aerosol. Forexample, such terminal apparatuses are disclosed in Japanese UnexaminedPatent Application Publication No. 2014-206291 and InternationalPublication No. 2016/181854.

SUMMARY

In one general aspect, the techniques disclosed here feature a displayapparatus including a display screen, and a controller that causes thedisplay screen to display a composite image in which a first imageacquired by imaging a space by a camera and a second image representingat least one type of aerosol existing in the space are combined. Theposition of the at least one type of aerosol as seen in a depthdirection in the first image is reflected in the second image.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a top view showing a space to which a non-contact sensingsystem according to an embodiment is applied;

FIG. 2 is a block diagram showing a configuration of a non-contactsensing system according to an embodiment;

FIG. 3 is a diagram showing an example of a sensor apparatus accordingto an embodiment;

FIG. 4 is a diagram showing an example of sensor data output from asensor apparatus according to an embodiment;

FIG. 5 is a block diagram showing a conditional expression fordetermining a control level in a non-contact sensing system according toan embodiment;

FIG. 6 is a diagram showing an example of a reference value databaseindicating a reference value for each substance;

FIG. 7 is a diagram for explaining a method of determining a contour ofaerosol by a non-contact sensing system according to an embodiment;

FIG. 8 is a diagram showing a representative value of a control levelfor each object in a space obtained by a non-contact sensing systemaccording to an embodiment;

FIG. 9 is a diagram showing an example of an image displayed on adisplay screen of a display apparatus according to an embodiment;

FIG. 10 is a sequence diagram showing an operation of a non-contactsensing system according to an embodiment;

FIG. 11 is a flowchart showing an operation of converting captured imagedata to a 3D database, which is one of operations performed by a noncontact sensing system according to an embodiment;

FIG. 12 is a flowchart showing an operation of converting sensor datainto a 3D database, which is one of operations performed by anon-contact sensing system according to an embodiment;

FIG. 13 is a diagram showing an example of a 3D database generated by anon-contact sensing system according to an embodiment;

FIG. 14 is a flowchart showing an operation of generating a leveldistribution, which is one of operations performed by a non-contactsensing system according to an embodiment;

FIG. 15 is a flowchart showing an operation of generating auxiliaryinformation, which is one of operations performed by a non-contactsensing system according to an embodiment;

FIG. 16 is a diagram showing another example of an image displayed on adisplay screen of a display apparatus according to an embodiment;

FIG. 17 is a diagram showing still another example of an image displayedon a display screen of a display apparatus according to an embodiment;

FIG. 18 is a diagram showing still another example of an image displayedon a display screen of a display apparatus according to an embodiment;

FIG. 19 is a diagram showing still another example of an image displayedon a display screen of a display apparatus according to an embodiment;

FIG. 20 is a diagram showing still another example of an image displayedon a display screen of a display apparatus according to an embodiment;

FIG. 21 is a diagram showing still another example of an image displayedon a display screen of a display apparatus according to an embodiment;and

FIG. 22 is a diagram showing a display apparatus integrally providedwith a non-contact sensing system according to an embodiment.

DETAILED DESCRIPTION Overview of the Present Disclosure

In an aspect, the present disclosure provides a display apparatusincluding a display screen, and a controller that causes the displayscreen to display a composite image in which a first image acquired byimaging a space by a camera and a second image representing at least onetype of aerosol existing in the space are combined. The position of theat least one type of aerosol as seen in a depth direction in the firstimage is reflected in the second image.

As described above, in the second image representing the aerosol, theposition of the aerosol in the depth direction is reflected. That is,not only the position of the aerosol in vertical and horizontaldirections represented by the first image but also the position of theaerosol as seen in the depth direction in the first image is displayedon the display screen. Thus, the position of the aerosol is presentedwith high accuracy.

For example, the first image may represent a two-dimensional space. Thecontroller may generate the second image by projecting three-dimensionalcoordinate data representing the position of the at least one type ofaerosol in the space onto the two-dimensional space. The controller maygenerate the composite image by combining the first image and the secondimage.

This allows it to accurately present the position of the aerosol n thetwo-dimensional space.

For example, the controller may acquire the three-dimensional coordinatedata from a sensor that acquires a position of the at least one type ofaerosol in the space. The controller may convert the first image into apseudo three-dimensional image. The controller may generate the secondimage by projecting the three-dimensional coordinate data into thetwo-dimensional space such that the pseudo three-dimensional image andthe three-dimensional coordinate data correspond to each other.

This allows it to accurately present the position of the aerosol in thepseudo three-dimensional space.

For example, the second image may include a contour representing aboundary of a region in which the at least one type of aerosol existsand distance information representing a distance in the space from areference position to a representative position of the region inside thecontour.

This makes it possible to display the boundary of the region in whichthe aerosol exists on the display screen, and also the position of theaerosol in the depth direction represented by the representativeposition. Thus, the position of the aerosol is displayed in a simplefashion that allows a user to easily understand the position by viewingthe display screen.

For example, the representative position may be a center of gravity of adensity distribution of the at least one type of aerosol in the regioninside the contour.

In this case, the representative position can be easily determined by acalculation based on the density distribution. In many cases, the closerto the center of the aerosol, the higher the density, and thus employingthe center of gravity of the density distribution as the representativeposition makes it possible to accurately present the position of theaerosol.

For example, the distance information may be a numerical valueindicating the distance.

The numerically displaying of the distance makes it possible to indicatethe position of the aerosol in a manner that allows a user to easilyunderstand the position.

For example, the distance information may be a color that ispredetermined according to the distance and is applied to the regioninside the contour.

This allows a user to distinguish distances by colors, that is, theposition of the aerosol is displayed in a manner that allows the user toeasily understand the position.

For example, the composite image may represent a three-dimensional modelincluding the space and a contour representing a boundary of a region inwhich the at least one type of aerosol exists.

By converting the composite image into the three-dimensional model, itbecomes possible to display the image as seen from various viewpoints onthe display screen. Thus, the position of the aerosol can be displayedin a manner that allows a user to easily understand the position.

For example, the second image may be a moving image including imagesthat are switched as time passes. Each of the images may correspond to adistance from a reference position in the space, and may include acontour indicating a boundary of a region, at the correspondingdistance, in which the at least one type of aerosol exists.

Thus, images representing the aerosol at respective distances aresequentially displayed and high-accuracy positions of aerosol arepresented.

For example, a density of the at least one type of aerosol may befurther reflected in the second image.

This allows it to present not only the position of the aerosol but alsothe density of the aerosol. As a result, a larger amount of informationand more types of information are presented to a user. This makes itpossible to more effectively assist in determining whether or not totake measures against the aerosol, such as performing ventilation.

For example, the second image may include level information indicating adensity level of the at least one type of aerosol.

By classifying the aerosol densities into levels, it becomes possible todisplay the aerosol density in a simple manner that allows a user toeasily understand the aerosol density by viewing the display screen.

For example, the at least one type of aerosol may include two or moretypes of aerosol, and the second image may represent the respective twoor more types of aerosol in different display modes.

Thus, even when two or more types of aerosol exist, it is possible todisplay the types in different modes depending on the types.

For example, the controller may further cause the display screen todisplay an image for warning a user in a case where a density of the atleast one type of aerosol is greater than a threshold value.

This can prompt a user to take countermeasures against the aerosol.

In an aspect, the present disclosure provides an image processingapparatus including an acquisition circuit that acquiresthree-dimensional coordinate data representing a position, in a space,of at least one type of aerosol existing in the space, and a processor.The processor generates a composite image in which a first imageacquired by imaging the space by a camera and a second imagerepresenting the at least one type of aerosol existing in the space arecombined based on the three-dimensional coordinate data. A position ofthe at least one type of aerosol as seen in a depth direction in thefirst image is reflected in the second image.

As described above, in the second image representing the aerosol, theposition of the aerosol in the depth direction is reflected. That is,not only the position of the aerosol in the vertical and horizontaldirections represented by the first image but also the position of theaerosol as seen in the depth direction in the first image is representedin the composite image displayed on the display screen. Thus, when thecomposite image generated by the image processing apparatus according tothe present aspect is displayed on the display screen, the position ofaerosol is accurately presented.

According to an aspect, the present disclosure provides a control methodof controlling a system, the system including a display apparatus and asensor including a light source that emits irradiation light toward atleast one type of object in a space and a photodetector that detectsreturn light returning from the at least one type of object, the sensoroutputting data representing a result of detection of the return lightby the photodetector, the control method including acquiring the datafrom the sensor, generating three-dimensional coordinate datarepresenting a position, in the space, of the at least one type ofobject based on the data, based on the three-dimensional coordinatedata, generating a composite image in which a first image and a secondimage are combined, the first image being obtained by imaging the spaceby a camera, the second image representing the at least one type ofobject existing in the space and reflecting a position of the at leastone type of object as seen in a depth direction in the first image, andcausing the display apparatus to display the composite image.

As described above, in the second image representing the object, theposition of the object in the depth direction is reflected. That is, notonly the position of the object in the vertical and horizontaldirections represented by the first image but also the position of theobject as seen in the depth direction in the first image is displayed onthe display screen. Thus, the position of the object is presented withhigh accuracy.

For example, the return light may be fluorescent light emitted by the atleast one type of object by being excited by the irradiation light, andthe generating of the composite image may include determining a type ofthe at least one type of object by analyzing the fluorescent light andreflecting the type in the second image.

This allows it to present not only the position of the object and alsothe type of the object. As a result, a larger amount of information andmore types of information are presented to a user, which makes itpossible to assist in determining whether or not to take measuresagainst the object, such as performing ventilation.

For example, the irradiation light may include a polarization component,and the generating of the composite image may include determining thetype of the at least one type of object based on a degree ofdepolarization of the polarization component included in the returnlight and reflecting the type in the second image.

This allows it to present not only the position of the object and alsothe type of the object. As a result, a larger amount of information andmore types of information are resented to a user, which makes itpossible to assist in determining whether or not to take measuresagainst the object, such as performing ventilation or disinfection.

For example, the three-dimensional coordinate data may be generatedusing coordinates of a position of the sensor in the space and arelative positional relationship between the sensor and the at least onetype of object calculated based on a difference between an irradiationlight emission time and a return light reception time.

In this case, the detection of the object and the measurement of thedistance to the detected object can be performed using the same lightsource and the same photodetector, and thus the configuration of thesensor apparatus can be simplified.

For example, the at least one type of object may be an organic substancestuck to an object existing in the space.

This makes it possible to detect a substance containing an organicsubstance such as vomit or pollen and accurately present the positionthereof.

For example, the at least one type of object may be aerosol existing inthe space.

That is, it is possible to detect a substance floating in the air suchas pollen or dust and accurately present the position thereof.

For example, the return light may be backscattered light generated as aresult of scattering of the irradiation light by the at least one typeof object.

This allows it to accurately detect aerosol.

According to an aspect, the present disclosure provides a non-transitorycomputer-readable storage medium storing a program for controlling asystem, the system including a display apparatus and a sensor includinga light source that emits irradiation light toward at least one type ofobject in a space and a photodetector that detects return lightreturning from the at least one type of object, the sensor outputtingdata representing a result of the detection of the return light by thephotodetector, the program, when executed by the computer, performingacquiring the data from the sensor, generating three-dimensionalcoordinate data representing a position, in the space, of the at leastone type of object based on the data, based on the three-dimensionalcoordinate data, generating a composite image in which a first image anda second image are combined, the first image being obtained by imagingthe space by a camera, the second image representing the at least onetype of object existing in the space and reflecting a position of the atleast one type of object as seen in a depth direction in the firstimage, and causing the display apparatus to display the composite image.

According to an aspect, the present disclosure provides acomputer-executable program for controlling a system, the systemincluding a display apparatus and a sensor including a light source thatemits irradiation light toward at least one type of object in a spaceand a photodetector that detects return light returning from the atleast one type of object, the sensor outputting data representing aresult of the detection of the return light by the photodetector, theprogram causing a computer to execute acquiring the data from thesensor, generating three-dimensional coordinate data representing aposition, in the space, of the at least one type of object based on thedata, based on the three-dimensional coordinate data; generating acomposite image in which a first image and a second image are combined,the first image being obtained by imaging the space by a camera, thesecond image representing the at least one type of object existing inthe space and reflecting a position of the at least one type of objectas seen in a depth direction in the first image, and causing the displayapparatus to display the composite image.

In the present disclosure, all or part of circuits, units, apparatuses,and elements, and all or part of functional blocks illustrated in thefigures may be implemented by one or more electronic circuits includinga semiconductor device, a semiconductor integrated circuit (IC), or anLSI (Large Scale Integration). The LSI or the IC may be integrated on asingle chip or may be realized by a combination of two or more chips.For example, functional blocks other than storage elements may beintegrated on a single chip. Note that the LSIs or ICs are calleddifferently, depending on the integration density, such as a system LSI,a VLSI (Very Large Scale Integration), or a ULSI (Ultra Large ScaleIntegration). A field programmable gate array (FPGA) capable of beingprogrammed after the LIS is produced, and a reconfigurable logic devicecapable of being reconfigured in terms of internal connections orcapable of being set up in terms of internal circuits segments may alsobe used for the same purpose.

Part or all of functions or operations of circuits, units, apparatuses,elements may be realized by performing a software process. In this case,software may be stored in a non-transitory storage medium. Thenon-transitory storage medium may be one of or a combination of a ROM,an optical disk, a hard disk drive, or the like. When the software isexecuted by a processing apparatus (a processor), a specific function isrealized by the processing apparatus (the processor) and a peripheraldevice. The system or the apparatus may include one or morenon-transitory storage media in which software is stored, a processor,and a hardware device such as an interface.

Specific embodiments are described below with reference to drawings.

Note that any embodiment described below is provided to illustrate ageneral or specific example. That is, in the following embodiments ofthe present disclosure, values, shapes, materials, constituent elements,locations of the constituent elements and manners of connecting theconstituent elements, steps, the order of steps, and the like aredescribed by way of example but not limitation. Among constituentelements described in the following embodiments, those constituentelements that are not described in independent claims are optional.

Note that each drawing is a schematic diagram, which does notnecessarily provide a strict description. For example, scales or thelike are not always consistent among drawings. Also note that, indrawings, substantially the same elements are denoted by the samereference numerals, and redundant descriptions of such elements areomitted or simplified descriptions are provided.

EMBODIMENTS Overview

A non-contact sensing system according to an embodiment captures animage of a space and detects an object existing in the space in anon-contact manner. The non-contact sensing system displays, on adisplay screen, a composite image in which a first image obtained byimaging a space and a second image representing a detected object arecombined. In the second image, the position of the detected object asseen in the depth direction in the first image is reflected.

First, the space to which the non-contact sensing system according tothe present embodiment is applied is described with reference to FIG. 1. FIG. 1 is a top view showing the space 95 to which the non-contactsensing system according to the present embodiment is applied.

The space 95 is, for example, a room in a building such as a house, anoffice, a nursing facility, or hospital. By way of example but notlimitation, the space 95 may be a closed space partitioned by walls, awindow, a door, a floor, a ceiling and/or the like, but the space 95 maybe an open space outside. Alternatively, the space 95 may be an internalspace of a mobile body such as a bus or an airplane.

As shown in FIG. 1 , a first object 90 to be detected by the non-contactsensing system exists in the space 95. Moe specifically, the firstobject 90 is aerosol floating in the space 95. Examples of the aerosolinclude dust, suspended particulate matter such as PM2.5, biologicalparticles such as pollen, or water microdroplets. Examples of biologicalparticles include molds or mites floating in the air. Examples ofaerosol may include a substance that is dynamically generated from ahuman body, such as a substance generated when coughing or sneezingoccurs. Still other examples of aerosol may include a substance which isa gas component in the air such as carbon dioxide (CO₂) to be detected.

The object to be detected is not limited to aerosol. For example, theobject may be organic dirt. Examples of organic dirt include foods orvomit stuck to an object such as a wall, a floor, or furniture existingin the space 95, which may not be suspended in the air.

2. Configuration

FIG. 2 is a block diagram illustrating a configuration of thenon-contact sensing system 10 according to an embodiment. As shown inFIG. 2 , the non-contact sensing system 10 includes a camera 20, a firstsensor 30, a second sensor 40, a third sensor 50, a computer 60, aserver apparatus 70, and a tablet terminal 80.

The configuration of the non-contact sensing system 10 is not limited tothe example shown in FIG. 2 . For example, the non-contact sensingsystem 10 may be configured so as to include only one of the firstsensor 30, the second sensor 40, and the third sensor 50. That is, thenumber of sensor apparatuses included in the non-contact sensing system10 may be only one, or may be two or more. The non-contact sensingsystem 10 may not include the computer 60 and the server apparatus 70.The non-contact sensing system 10 may include, instead of the tabletterminal 80, a display connected to the computer 60.

Although not shown in FIG. 2 , the camera 20, the first sensor 30, thesecond sensor 40, the third sensor, the server apparatus 70, and thetablet terminal 80 each include a communication interface such thatvarious kinds of data and information are allowed to be transmitted andreceived via the communication interface.

The details of the constituent elements of the non-contact sensingsystem 10 are described below with reference to FIG. 2 .

2.1 Camera

The camera 20 captures an image of the space 95 thereby generating acaptured image. The captured image is an example of the first imagegenerated by the camera 20 by imaging the space 95. The camera 20 maybe, for example, a fixed-point camera fixed at a position that allows itfor the camera 20 to image the space 95. However, the camera 20 is notlimited to the fixed-point type. For example, the camera 20 may be amovable camera which is movable in terms of at least one of a shootingposition and a shooting direction. The camera 20 may generate two ormore captured images by imaging the space 95 from two or moreviewpoints. The camera 20 transmits the captured image data obtained bycapturing the image to the computer 60. The camera 20 may be a visiblelight camera that captures an image of a space visible to humans.

2.2 Sensor Apparatus

The first sensor 30, the second sensor 40, and the third sensor 50 areeach an example of the sensor apparatus that contactlessly detects anobject to be detected. That is, the non-contact sensing system 10according to the present embodiment includes three sensor apparatusesfor respectively detecting particular types of objects. For example, thefirst object 90 shown in FIG. 2 is pollen, which is detected by thefirst sensor 30. The second object 92 is dust, which is detected by thesecond sensor 40. The third object 94 is an organic dirt, which isdetected by the third sensor 50. Each of the first sensor 30, the secondsensor 40, and the third sensor 50 is a non-contact sensor apparatususing, for example, LIDAR (Laser Imaging Detection and Ranging).

FIG. 3 is a diagram showing the first sensor 30, which is an example ofthe sensor apparatus according to the present embodiment. In the presentembodiment, the first sensor 30 is an autonomous mobile sensorapparatus. Note that the second sensor 40 and the third sensor 50 eachhave the same configuration as that of the first sensor 30.

As shown in FIG. 3 , the first sensor 30 is capable of moving on a floorsurface 96 in the space 95. The first sensor 30 emits irradiation lightL1 at a predetermined position on the floor surface 96 and detectsreturn light L2 returning from the first object 90. The first sensor 30measures the distance to the first object 90 based on a time differencebetween a time pf the emission of the irradiation light L1 and a time ofthe detection of the return light L2. The first sensor 30 also measuresthe density of the first object 90 based on the intensity of the returnlight L2.

More specifically, as shown in FIG. 2 , the first sensor 30 includes alight source 32, a photodetector 34, and a signal processing circuit 36.

The light source 32 is a light source that emits the irradiation lightL1 toward the first object 90 in the space 95. The light source 32 maybe, for example, an LED (Light Emitting Diode) or a laser device. Theirradiation light L1 emitted by the light source 32 has a wavelengthcomponent for exciting the first object 90. More specifically, theirradiation light L1 may be light having a peak wavelength in a rangefrom a wavelength equal to or larger than 220 nm to a wavelength equalto or smaller than 550 nm. The irradiation light L1 may be, for example,pulsed light.

The photodetector 34 is a photodetector that detects the return light L2returning from the first object 90. The return light L2 detected by thephotodetector 34 is fluorescent light emitted by the first object 90when it is excited by the irradiation light L1 emitted from the lightsource 32. The fluorescent light contains more long-wavelengthcomponents than the irradiation light L1 contains. The photodetector 34may be, for example, a photodiode sensitive to the wavelength componentsof the fluorescent light. The photodetector 34 outputs an output signalcorresponding to the intensity of the received fluorescent light to thesignal processing circuit 36. The output signal is, for example, anelectric signal whose signal strength increases as the intensity of thereceived fluorescent light increases.

The signal processing circuit 36 determines the distance to the firstobject 90 and the density of the first object 90 by processing theoutput signal output from the photodetector 34. As shown in FIG. 2 , thesignal processing circuit 36 includes a position information acquisitionunit 37 and a density information acquisition unit 38.

The position information acquisition unit 37 acquires positioninformation indicating a three-dimensional position of the first object90 in the space 95. The position information includes informationindicating the distance to the first object 90 and informationindicating the direction thereto. For example, the position informationacquisition unit 37 calculates the distance by a TOF (Time Of Flight)method. More specifically, the position information acquisition unit 37acquires distance information based on the time difference between thetime of the emission of the irradiation light L1 from the light source32 and the time of the detection of fluorescent light by thephotodetector 34. The distance information includes informationindicating the distance ri to the first object 90 and informationindicating the direction, represented by the horizontal angle (pi andthe vertical angle θi, in which the first object 90 is detected. Notethat the direction in which the first object 90 is detected is the sameas the direction in which the light source 32 emits the irradiationlight L1.

The density information acquisition unit 38 acquires density informationindicating the density of the first object 90. More specifically, thedensity information acquisition unit 38 determines the density of thefirst object 90 according to the signal strength of the output signal.For example, when the signal strength is denoted as Si, the density Diis calculated according to formula (1) shown below.Di=α×Si  (1)

In formula (1), α is a constant, subscript “i” of each of Di, Si, ri,φi, and θi indicates the data number of the sensor data. Note that themethod of calculating the density Di by the density informationacquisition unit 38 is not limited to the example described above. Forexample, the density information acquisition unit 38 may use a signalobtained after removing noise components from the original output signalinstead of the original output signal.

The signal processing circuit 36 may determine the type of the firstobject 90 by analyzing the fluorescent light. More specifically, thesignal processing circuit 36 may determine the type of the first object90 based on a combination of the wavelength of the irradiation light andthe wavelength of the fluorescent light. For example, in the firstsensor 30, the light source 32 may emit two or more irradiation lightbeams corresponding to two or more excitation wavelengths, and thephotodetector 34 may detect two or more pieces of fluorescent lightcorresponding to two or more received light wavelengths. To accuratelydetermine the type of the first object 90 that generates fluorescentlight, the signal processing circuit 36 may generate a so-calledfluorescence fingerprint in the form of a three-dimensional matrix ofthe excitation wavelength, the received light wavelength, and thereceived light intensity.

The signal processing circuit 36 outputs the density informationindicating the determined density Di and the position information to thecomputer 60 as sensor data.

The first sensor 30 and the computer 60 are, for example, wirelesslyconnected such that data can be transmitted and received between them.The first sensor 30 performs wireless communication according to awireless communication standard such as Wi-Fi (registered trademark),Bluetooth (registered trademark), or ZigBee (registered trademark). Theconnection between the first sensor 30 and the computer 60 may berealized via a wire.

The second sensor 40 emits irradiation light toward the second object 92and detects return light returning from the second object 92 therebydetecting the second object 92. In the present embodiment, the secondobject is a substance that does not emit fluorescent light. An exampleof the second object is dust.

The second sensor 40 includes a light source 42, a photodetector 44, anda signal processing circuit 46. The light source 42, the photodetector44, and the signal processing circuit 46 respectively correspond to thelight source 32, the photodetector 34, and the signal processing circuit36 in the first sensor 30.

The light source 42 is a light source that emits irradiation lighttoward the second object 92. The light source 42 may be, for example, anLED or a laser device. The irradiation light emitted by the light source42 does not need to excite the second object 92. Therefore, theirradiation light is allowed to have a wavelength component selectedfrom a wide wavelength band. More specifically, the irradiation lightemitted from the light source 42 may be light having a peak wavelengthin a range from a value equal to or greater than 300 nm to a value equalto or smaller than 1300 nm. That is, the irradiation light may beultraviolet light, visible light, or near infrared light. Theirradiation light may be, for example, pulsed light.

The photodetector 44 is a photodetector that detects return light L2returning from the second object 92. The return light detected by thephotodetector 44 is backscattered light generated when the irradiationlight emitted from the light source 42 is scattered by the second object92. For example, the backscattered light is Mie scattering light. Thebackscattered light has the same wavelength components as those of theirradiation light. The photodetector 44 may be, for example, aphotodiode sensitive to the wavelength component of the irradiationlight. The photodetector 44 outputs an output signal according to theintensity of the received backscattered light to the signal processingcircuit 46. The output signal is, for example, an electric signal whosesignal strength increases as the intensity of the received backscatteredlight increases.

In the present embodiment, the irradiation light emitted by the lightsource 42 may include a predetermined polarization component. The signalprocessing circuit 46 may determine the type of the second object 92based on a depolarization degree of the polarization component includedin the return light. The polarization component is, for example,linearly polarized light, but may be circularly polarized light orelliptically polarized light. When the second object 92 is irradiatedwith the irradiation light including the polarization component, thebackscattered light returning from the second object 92 has adepolarization degree varying depending on the shape of the secondobject 92.

More specifically, when the second object 92 is a cluster of sphericalparticles, the polarization state is retained in its backscatteredlight. That is, the polarization state of the backscattered light is thesame as the polarization state of the irradiation light. When the secondobject 92 is a cluster of non-spherical particles, the plane ofpolarization changes depending on the shape of the particles. This makesit possible for the signal processing circuit 46 to determine the typeof the second object based on the depolarization degree of thebackscattered light. For example, the depolarization degree of yellowsand is about 10%, and the depolarization degree of pollen is in a rangefrom about 1% to about 4%.

The signal processing circuit 46 determines the distance to the secondobject 92 and the density of the second object 92 by processing theoutput signal output from the photodetector 44. As shown in FIG. 2 , thesignal processing circuit 46 includes a position information acquisitionunit 47 and a density information acquisition unit 48. Specificoperations of determining the distance and the density are the same asthose of the signal processing circuit 36 of the first sensor 30.

The third sensor 50 emits irradiation light toward the third object 94and detects return light returning from the third object 94 therebydetecting the third object 94. In the present embodiment, the thirdobject 94 is an organic dirt that emits fluorescent light whenirradiated with excitation light.

The third sensor 50 includes a light source 52, a photodetector 54, anda signal processing circuit 56. The signal processing circuit 56includes a position information acquisition unit 57 and a densityinformation acquisition unit 58.

The light source 52, the photodetector 54, and the signal processingcircuit 56 respectively correspond to the light source 32, thephotodetector 34, and the signal processing circuit 36 of the firstsensor 30. The first sensor 30 and the third sensor 50 are differentfrom each other in directions in which the respective light sources emitthe irradiation light. More specifically, the light source 32 emits theirradiation light toward the air in the space 95, while the light source52 emits the irradiation light toward the floor surface or the wallsurface of the space 95. The operations of the light source 52, thephotodetector 54, and the signal processing circuit 56 are respectivelythe same as those of the light source 32, the photodetector 34, and thesignal processing circuit 36.

The first sensor 30, the second sensor 40, and the third sensor 50 eachdetect an object located in the direction in which the irradiation lightis emitted. In a case where there are two or more objects in theemission direction of the irradiation light, the return light returns atdifferent times depending on the positions of the objects. Therefore,two or more objects located in the emission direction of the irradiationlight are detected at a time based on the time when the return light isreceived. Note that in a case where there is no object in the emissiondirection of the irradiation light, no return light is detected.Therefore, when no return light is detected, it is determined that noobject exists on the path of the irradiation light. Each of the firstsensor 30, the second sensor 40, and the third sensor 50 transmits adetection result as sensor data to the computer 60.

FIG. 4 illustrates an example of a database including sensor data outputfrom the sensor apparatus according to the present embodiment. Thedatabase shown in FIG. 4 is managed by the processor 64 of the computer60 and stored in the memory 66.

As shown in FIG. 4 , in the database, the substance name Mi, the sensordata, and the sensor reference position are associated with each otherfor each data number No. i. The sensor data includes data in terms ofthe density Di, distance ri, horizontal angle φi, and vertical angle θi.

One data number No. i is assigned to each sensor data received by thecomputer 60. More specifically, for example, the processor 64 assignsthe data numbers in ascending order in which the communication interface62 receives the sensor data.

The substance name Mi is information indicating the type of the objectto be detected. In the present embodiment, the types of objectscorrespond to the respective sensor apparatuses. Therefore, theprocessor 64 can determine the substance name Mi corresponding to thesensor data by determining the sender of the sensor data received viathe communication interface 62. For example, in the example shown inFIG. 4 , the sensor data of data number 1 is data transmitted from thefirst sensor 30 that detects pollen.

The density Di is a value calculated according to formula (1) describedabove. The signal processing circuits 36, 46, and 56 of the respectivesensor apparatuses perform the calculation based on the signal strengthSi.

The distance ri, the horizontal angle φi, and the vertical angle θiindicate the three-dimensional position of the object obtained by usingLIDAR. The position data obtained by LIDAR is represented in the polarcoordinate system, and thus, in the present embodiment, the computer 60performs a coordinate conversion on the position data into thethree-dimensional orthogonal coordinate system. Details of thecoordinate conversion will be described later.

The sensor reference position is, for example, the installation positionof the sensor apparatus that is one of the first sensor 30, the secondsensor 40, and the third sensor 50 and that has transmitted the sensordata. In a case where the sensor apparatus is fixed, the sensorreference position does not change. In a case where the sensor apparatusis movable, the sensor reference position is the position of the sensorapparatus at the time when the detection process is performed, and morespecifically, when the irradiation light is output or the return lightis received. The reference directions of the horizontal angle φi and thevertical angle θi transmitted by the respective sensors, that is, thedirections in which φi=0 and θi=0, are set in advance in the samedirections among the sensors.

2.3 Computer

The computer 60 is an example of the image processing apparatus, andincludes the communication interface 62, the processor 64, and thememory 66, as shown in FIG. 2 .

The communication interface 62 transmits and receives data incommunicating with each device included in the non-contact sensingsystem 10. The communication with each device may be performedwirelessly based on a wireless communication standard such as Wi-Fi(registered trademark), Bluetooth (registered trademark), or ZigBee(registered trademark), or may be performed via wire.

The communication interface 62 is an example of the acquisition circuitthat acquires three-dimensional coordinate data. The communicationinterface 62 obtains sensor data from each of the first sensor 30, thesecond sensor 40, and the third sensor 50 by communicating with each ofthe first sensor 30, the second sensor 40, and the third sensor 50. Thesensor data includes position information, which is an example ofthree-dimensional coordinate data representing the position of at leastone type of object in the space 95. The sensor data further includesdensity information.

The three-dimensional coordinate data is generated using coordinates ofthe position of the sensor apparatus in the space 95 and a relativepositional relationship between the sensor apparatus and an objectcalculated based on the difference between the irradiation lightemission time and the return light reception time. The relativepositional relationship corresponds to the distance ri shown in FIG. 4 .The coordinates of the sensor apparatus in the space 95 correspond tothe coordinates (x0, y0, z0) indicating the reference position shown inFIG. 4 .

The communication interface 62 also acquires captured image data fromthe camera 20 by communicating with the camera 20. The communicationinterface 62 may transmit a control signal including an image capturinginstruction or a sensing instruction to at least one of the camera 20,the first sensor 30, the second sensor 40, and the third sensor 50. Thecommunication interface 62 further communicates with the serverapparatus 70 to transmit level distribution information corresponding tothe density distribution of the object to the server apparatus 70. Thecommunication interface 62 communicates with the tablet terminal 80 totransmit composite image data to the tablet terminal 80.

The processor 64 generates a composite image based on the sensor dataacquired via the communication interface 62. The composite image is animage obtained by combining a captured image representing the space 95captured by the camera 20 and an object image. The object image is anexample of the second image representing at least one type of objectexisting in the space 95.

In the present embodiment, the processor 64 generates the densitydistribution of the object in the space 95 based on the sensor data.More specifically, the processor 64 generates a three-dimensionaldistribution of density such that the space 95 is represented bycoordinates in a three-dimensional orthogonal coordinate system and adensity is associated with each coordinate point. In FIG. 1 , an x-axis,a y-axis and a z-axis shown are three axes of the three-dimensionalorthogonal coordinate system. The x-axis and the y-axis are two axesparallel to the floor surface of the space 95, and the z-axis is an axisperpendicular to the floor surface. Note that the setting of the threeaxes is not limited to this example.

More specifically, the processor 64 generates a level distribution whichis an example of the density distribution of the object. The leveldistribution is a distribution of the control level Ci determined basedon the density information. In the present embodiment, the density Di isclassified into two or more levels according to magnitudes. The controllevel Ci is given by one of level values into which the density Diindicated by the density information is classified. For example, theprocessor 64 determines the control level Ci according to a conditionalexpression shown in FIG. 5 . FIG. 5 shows the conditional expression fordetermining the control level Ci in the non-contact sensing system 10according to the present embodiment. The conditional expression isstored, for example, in the memory 66.

As shown in FIG. 5 , the control level Ci is represented one of fivelevels from 1 to 5. The processor 64 determines the control level Cibased on the relationship between the density Di and the reference valueLm. The reference value Lm is, as shown in FIG. 6 , a valuepredetermined for each type of object. FIG. 6 shows an example of areference value database indicating a reference value for eachsubstance. The reference value database is stored, for example, in thememory 66. The number of levels of the control level Ci is not limitedto 5, but it may be 2, 3, or 4, or may be 6 or greater. In theconditional expression shown in FIG. 5 , the value of the coefficient(for example, “0.4”) by which the reference value Lm is multiplied ismerely an example.

In the present embodiment, processor 64 further determines a contour ofan object based on the generated three-dimensional distribution.Furthermore, the processor 64 determines a particular position insidethe determined contour as a representative position. The object imageincludes the determined contour and the representative position.

For example, the processor 64 determines the contour of the object basedon the densities Di at respective coordinate points. More specifically,the processor 64 determines the contour of the aerosol existing in thespace 95 based on the control level Ci calculated based on the densitiesDi at the respective coordinate points.

Referring to FIG. 7 , an example of a method is described below fordetermining a contour of aerosol by the non-contact sensing system 10according to the present embodiment. In the example of the method shownin FIG. 7 , for simplification, a contour is determined within atwo-dimensional level distribution in a plane defined by the x-axis andthe y-axis. Note that a contour of a three-dimensional leveldistribution can also be determined in a similar manner.

As shown in FIG. 7 , the control level Ci is calculated for eachcoordinate set of an x-coordinate and a y-coordinate. For example, theprocessor 64 determines a region in which the control level Ci is equalto or higher than a set value, and determines a contour of this regionas the contour of the aerosol. For example, in a case where the setvalue is “2”, the processor 64 determines a contour 90 a of a regionwhere the control level Ci is equal to or higher than “2” as the contourof the aerosol. Note that, in FIG. 7 , areas where the control level Ciis equal to or higher than “2” are shaded with dots. In the exampleshown in FIG. 7 , aerosols is detected at two locations in the space.

Note that the set value for determining the contour may be variable. Forexample, when the set value is increased, an area where the aerosoldensity is higher is determined as a region where aerosol exists.Alternatively, when the set value is reduced, a region where the aerosoldensity is higher than a level lower than a level indicated by thecurrent set value is determined as a region where aerosol exists.

Note that the processor 64 may use two or more setting values indetermining the contour such that contours are determined for therespective setting values. For example, in the example shown in FIG. 7 ,a contour 90 a corresponding to a setting value of “2” while a contour90 b corresponding to a setting value of “3” are determined. The contour90 a is the outermost contour among the determined contours, andindicates the existence boundary of the aerosol. The contour 90 b is acontour indicating a region where the density of the aerosol is higherin the region where aerosol exists. As described above, the densitydifference of the aerosol can be represented by the contours in theregion where the aerosol exists.

The representative position of an area inside a contour is given by acenter of gravity of an aerosol density distribution inside the counter.More specifically, the processor 64 determines the center of gravitybased on the control level Ci for each of coordinate sets inside thecontour. For example, when the coordinates of the center of gravity isdenoted by (Xc, Yc, Zc), the processor 64 determines the coordinates ofthe center of gravity according to the following formula (2).Xc=Σ(Di×Xi)/Σ(Di)Yc=Σ(Di×Yi)/Σ(Di)Zc=Σ(Di×Zi)/Σ(Di)  (2)

In formula (2), Σ( ) is an arithmetic symbol representing the sum ofterms in ( ), and i corresponds to a coordinate point located within thearea surrounded by the determined contour.

The representative position may be the center of gravity of athree-dimensional graphical figure having the determined contour on theouter circumference.

The memory 66 is a storage apparatus for storing captured image data andsensor data. The memory 66 also stores a program executed by theprocessor 64 and parameters used in executing the program, and the like.The memory 66 also serves as a program execution area for use by theprocessor 64. The memory 66 may include, for example, a nonvolatilememory such as an HDD (Hard Disk Drive) or a semiconductor memory, and avolatile memory such as a RAM (Random Access Memory).

2.4 Server Apparatus

The server apparatus 70 receives the level distribution informationtransmitted from the computer 60 and performs a process using thereceived level distribution information. More specifically, the serverapparatus 70 warns a person who uses the space 95 based on the leveldistribution information. For example, the server apparatus 70 generatesa warning image which is an image for warning, and transmits thegenerated warning image to the tablet terminal 80.

For example, the server apparatus 70 determines whether or not thedetected density of at least one type of object is greater than athreshold value, More specifically, the server apparatus 70 determineswhether or not the representative control level C in the space 95 ishigher than a threshold value. When the server apparatus 70 determinesthat the representative control level C is higher than the thresholdvalue, the server apparatus 70 generates a warning image. The thresholdvalue is by way of example but not limitation a predetermined fixedvalue. For example, the threshold value may be appropriately updated bymachine learning.

The representative control level C is calculated, for example, based onthe representative value Cm of the control level for each object. Therepresentative value Cm is a value representing the control level of thecorresponding object, and is given by, for example, the maximum value ofthe control levels in the level distribution of the correspondingobject. The server apparatus 70 calculates the representative value Cmfor each object based on the level distribution.

FIG. 8 shows a representative value Cm of the control level for eachobject in the space 95 obtained by the non-contact sensing system 10according to the present embodiment. The server apparatus 70 calculatesthe representative control level C by averaging the representativevalues of the objects. For example, in the example shown in FIG. 8 , therepresentative control level C is “3.8”.

Note that the representative control level C may not be given by theaverage value of the representative values Cm. For example, therepresentative control level C may be a sum of weighted representativevalues Cm. For example, when the weight is 1 for pollen and dust, theweights for CO2, water, and surface organic dirt may be 0.3, 0.1, and0.1, respectively. The values of the weights are not limited to thesevalues. The weights may be changeable based on an instruction given by auser or the like.

Furthermore, the server apparatus 70 may control an air conditionerinstalled in the space 95. The server apparatus 70 may give preventiveadvice for suppressing an increase in the density pollen, dust, or thelike. The preventive advice is, for example, an instruction to prompt auser to ventilate the space 95 or an instruction to drive a device suchas an air purifier installed in the space 95. The server apparatus 70outputs image data or voice/sound data including preventive advice tothe tablet terminal 80. For example, the server apparatus 70 acquiresinformation regarding alerting or preventive advice by referring tometeorological observation data or the like. Furthermore, the serverapparatus 70 may generate information regarding a warning or preventiveadvice by performing machine learning based on a temporal change in thedensity or the control level.

2.5 Tablet Terminal

The tablet terminal 80 is a portable information processing terminal.The tablet terminal 80 may be a multifunctional information terminalsuch as a tablet PC or a smartphone, or may be an information terminaldedicated to the non-contact sensing system 10. As shown in FIG. 2 , thetablet terminal 80 is an example of the display apparatus including adisplay screen 82 and a controller 84.

The display screen 82 displays a composite image. The display screen 82is, for example, a liquid crystal display panel, but is not limited tothis. For example, the display screen 82 may be an emissive displaypanel using an organic EL (Electroluminescence) element. The displayscreen 82 may be, for example, a touch panel display which is capable ofaccepting an input from a user.

The controller 84 performs control such that a composite image isdisplayed on the display screen 82. The controller 84 includes, forexample, a non-volatile memory in which a program is stored, a volatilememory serving as a temporary storage area for use in executing theprogram, an input/output port, a processor that executes the program,and/or the like.

In the present embodiment, the controller 84 acquires composite imagedata transmitted from the computer 60 and displays a composite image onthe display screen 82 based on the acquired composite image data. Forexample, the controller 84 performs control such that a composite image100 shown in FIG. 9 is displayed on the display screen 82.

FIG. 9 illustrates an example of a display image on the display screen82 of the tablet terminal 80 which is an example of the displayapparatus according to the present embodiment. As shown in FIG. 9 , thecomposite image 100 is displayed on the display screen 82.

The composite image 100 is an image in which a captured image 101 and anaerosol image 102 are combined. The composite image 100 is, for example,a still image.

The captured image 101 represents the space 95 imaged by the camera 20.The captured image 101 is an example of the first image. The capturedimage 101 is obtained, by way of example but Plot limitation, by imagingthe space 95 from a horizontal direction. The captured image 101 may be,for example, an image obtained by imaging the space 95 from above. Inthis case, the captured image 101 corresponds to the top view shown inFIG. 1 .

The aerosol image 102 is an example of the object image representing atleast one type of object existing in the space 95. For example, theaerosol image 102 represents pollen which is an example of aerosol. Theaerosol image 102 reflects the position of at least one type of objectin the depth direction in the captured image 101. The aerosol image 102is an example of the second image.

As shown in FIG. 9 , the aerosol image 102 includes a contour 102 a anddistance information 102 b. The contour 102 a represents, for example, aboundary of a region in which the first object 90 detected by the firstsensor 30 exists. The distance information 102 b is a numerical valueindicating the distance from the reference position to therepresentative position in the contour 102 a.

The reference position is a position defined in the space 95. Forexample, the reference position is the installation position of thecamera 20. Alternatively, the reference position may be the position ofa person or a device such as an air purifier existing in the space 95.

The aerosol image 102 may include information related to the density ofthe aerosol, as will be described in detail later with reference toother examples shown in FIGS. 16 to 19 . More specifically, the aerosolimage 102 may include level information indicating the control level Ciof the density of aerosol. In a case where two or more types of aerosolare detected, the aerosol image may represent the two or more types ofaerosol in different display modes. When the density of the aerosol ishigher than the threshold value, a warning image for warning a user maybe displayed on the display screen 82.

3. Operation

Next, an operation of the non-contact sensing system 10 according to anembodiment is described with reference to FIGS. 10 to 15 .

FIG. 10 is a sequence diagram illustrating the operation of thenon-contact sensing system 10 according to the present embodiment.

As shown in FIG. 10 , first, the camera 20 images the space 95 (S10).The camera 20 transmits obtained captured image data to the computer 60(S12).

The first sensor 30 performs a detection process for detecting the firstobject 90 (S14). More specifically, in the first sensor 30, the lightsource 32 emits irradiation light toward the first target object 90, andthe photodetector 34 detects return light returning from the firsttarget object 90. The signal processing circuit 36 generates sensor dataincluding data indicating a distance of the first object 90 and adensity of the first object 90 based on a signal strength of the returnlight. The first sensor 30 transmits the generated sensor data to thecomputer 60 (S16).

The second sensor 40 performs a detection process for detecting thesecond object 92 (S18). More specifically, in the second sensor 40, thelight source 42 emits irradiation light toward the second object 92, andthe photodetector 44 detects return light returning from the secondobject 92. The signal processing circuit 46 generates sensor dataincluding data indicating a distance of the second object 92 and adensity of the second object 92 based on a signal strength of the returnlight. The second sensor 40 transmits the generated sensor data to thecomputer 60 (S20).

The third sensor 50 performs a detection process for detecting the thirdobject 94 (S22). More specifically, in the third sensor 50, the lightsource 52 emits irradiation light toward the third object 94, and thephotodetector 54 detects return light returning from the third subject94. The signal processing circuit 56 generates sensor data includingdata indicating a distance of the third object 94 and a density of thethird object 94 based on a signal strength of the return light. Thethird sensor 50 transmits the generated sensor data to the computer 60(S24).

Note that among the above-described processes, i.e., the image capturing(S10) by the camera 20, the detection process (S14) by the first sensor30, the detection process (S18) by the second sensor 40, and thedetection process (S22) by the third sensor 50, any one of theseprocesses may be performed first or these processes may be performed inparallel. The image capturing (S10) and the detection processes (S14,S18, and S22) may be performed at timings based on an instruction givenby the computer 60, the server apparatus 70, or the like. Each of thedevices (the camera, the sensors) transmits the captured image data orthe sensor data when the captured image data or the sensor data isobtained. Alternatively, each of the devices may transmit the capturedimage data or the sensor data when receiving a request from the computer60.

When the computer 60 receives the captured image data and each sensordata, the computer 60 generating a 3D database based on the receivedcaptured image data and each sensor data (S26). More specifically, theprocessor 64 of the computer 60 converts the two-dimensional capturedimage into a pseudo three-dimensional image. Furthermore, the processor64 transforms the sensor data from the polar coordinate system to athree-dimensional orthogonal coordinate system.

FIG. 11 is a flowchart showing an operation of converting captured imagedata to a 3D database, which is one of operations performed by thenon-contact sensing system 10 according to the present embodiment. Notethat FIG. 11 shows an example of a detailed operation in step S26 shownin FIG. 10 .

As shown in FIG. 11 , the processor 64 acquires captured image data viathe communication interface 62 (S102). A captured image included in thecaptured image data is a two-dimensional image. The processor 64converts the two-dimensional captured image into a pseudothree-dimensional image by using a generally known method for convertinga two-dimensional image into a pseudo three-dimensional image (S104).

Note that the captured image data may include a distance imageindicating distances to walls, a floor, and a ceiling that form thespace 95, and distances to a person and furniture located in the space95. The captured image data may include two or more captured imagescaptured at different viewpoints. The processor 64 may generate athree-dimensional image by using a captured image and a distance imageor by using a two or more captured images. This makes it possible toenhance the likelihood of the three-dimensional image.

FIG. 12 is a flowchart showing an operation of converting sensor datainto a 3D database, which is one of operations performed by thenon-contact sensing system 10 according to the present embodiment. Notethat FIG. 12 shows an example of a detailed operation in step S26 shownin FIG. 10 .

As shown in FIG. 12 , the processor 64 acquires sensor data from thedatabase stored in the memory 66 (S112). More specifically, theprocessor 64 acquires a distance ri, a horizontal angle φi, a verticalangle θi, and a substance name Mi. The processor 64 converts theacquired sensor data into spatial coordinates, that is, coordinates in athree-dimensional orthogonal coordinate system, according to a followingformula (3) (S114).Xi=x0+ri×cos θi·sin φiYi=y0+ri×cos θi·cos φiZi=z0+ri×sin θi  (3)

Either of the conversion of the captured image data into the pseudothree-dimensional image shown in FIG. 11 and the conversion of thesensor data into the three-dimensional data shown in FIG. 12 may beperformed first, or may be performed simultaneously. By performing thethree-dimensionalization on the sensor data, the spatial coordinates(Xi, Yi, Zi) represented in the three-dimensional orthogonal coordinatesystem are associated with the data number No. i, as shown in FIG. 13 .

FIG. 13 shows an example of a 3D database generated by the non-contactsensing system 10 according to the present embodiment. As shown in FIG.13 , a substance name Mi, a density Di, a control level Ci, and spatialcoordinates (Xi, Yi, Zi) are associated with each data number No. i.

After the 3D database is generated, as shown in FIG. 10 , the computer60 generates a level distribution based on the generated 3D database(S28). The computer 60 transmits level distribution informationindicating the generated level distribution to the server apparatus 70(330).

The process of generating the level distribution is described in furtherdetail below with reference to FIG. 14 . FIG. 14 is a flowchart showingthe process of generating the level distribution, which is one ofoperations performed by the non-contact sensing system 10 according tothe present embodiment. Note that FIG. 14 shows an example of a detailedoperation in step S28 shown in FIG. 10 .

As shown in FIG. 14 , first, the processor 64 acquires the densityinformation and the spatial coordinates (Xi, Yi, Zi) by reading themfrom the memory 66 (3122). Next, the processor 64 determines the controllevel Ci based on a comparison with the reference value Lm for eachsubstance and generates the level distribution (S124). Next, theprocessor 64 determines a contour and a representative position in thecontour based on the generated level distribution (S126). Note that theprocess of determining the contour and the representative position isperformed in the manner described above with reference to FIG. 7 .

After the level distribution is generated, the computer 60 generates acomposite image as shown in FIG. 10 (S32). More specifically, thecomputer 60 combines the contour and the distance information with thecaptured image by mapping the level distribution on the captured image.The computer 60 transmits the composite image data to the tabletterminal 80 (S34).

The image including the contour and distance information is an exampleof the second image generated by projecting three-dimensional coordinatedata representing the position in the space of at least one type ofaerosol in the two-dimensional space represented by the captured image.A specific example of the image including the contour and the distanceinformation is the aerosol image 102 shown in FIG. 9 .

More specifically, the computer 60 generates the image including thecontour and the distance information by projecting the three-dimensionalcoordinate data representing the position of at least one type ofaerosol in the two-dimensional space represented by the captured image.For example, the computer 60 extends the captured image to apseudo-three-dimensional image and performs a projection such that theextended three-dimensional image and the three-dimensional coordinatedata correspond to each other, thereby generating the image includingthe contour and the distance information. The three-dimensional imageand the three-dimensional coordinate data are made to correspond to eachother by arranging the three-dimensional image and the three-dimensionalcoordinate data such that the origin and the three axes in thethree-dimensional coordinate system in which the three-dimensional imageis represented coincide with the origin and the three axes of thethree-dimensional coordinate system in which the three-dimensionalcoordinate data are represented. The computer 60 generates a compositeimage by combining a captured image with an image including a contourand distance information.

The server apparatus 70 acquires auxiliary information based on thelevel distribution information transmitted from the computer 60 (336).The auxiliary information is, for example, information includinginformation indicating a warning or preventive advice. The serverapparatus 70 transmits the acquired auxiliary information to the tabletterminal 88 (338).

Next, details of a process of generating the auxiliary information aredescribed with reference to FIG. 15 . FIG. 15 is a flowchart showing aprocess of generating the auxiliary information, which is one ofoperations performed by the non-contact sensing system 10 according tothe present embodiment. Note that the process shown in FIG. 15 is anexample of the detailed operation in step S36 shown in FIG. 10 .

As shown in FIG. 15 , first, the server apparatus 70 determines therepresentative value Cm of the control level for each object in thespace 95 (S132). Next, the server apparatus 70 determines therepresentative control level C in the space 95 (S134). The specificmethod of determining the representative control level C is the same asdescribed above with reference to FIG. 8 .

Next, the server apparatus 70 compares the representative control levelC with the threshold value (S136). When the representative control levelC is higher than the threshold value (Yes in S136), the server apparatus70 generates a warning image (S138). Instead of the warning image,preventive advice may be generated. In a case where the representativecontrol level C is lower than or equal to the threshold value (No inS136), the auxiliary information generation process is ended.

Although the server apparatus 70 compares the representative controllevel C with the threshold value in the example described above, theserver apparatus 70 may compare the representative value Cm of thecontrol level for each object with the threshold value. In this case,the server apparatus 70 does not need to determine the representativecontrol level C. For example, when at least one of the representativevalues Cm of the control levels of the respective objects such aspollen, dust, or the like is larger than the threshold value, the serverapparatus 70 may generate a warning image.

Finally, as shown in FIG. 10 , the tablet terminal 80 acquires thecomposite image data transmitted from the computer 60 and the auxiliaryinformation transmitted from the server apparatus 70, and displays thecomposite image on the display screen 82 (S40). The composite imagedisplayed on the display screen 82 may not include the auxiliaryinformation. The composite image 100 shown in FIG. 9 is an example ofthe composite image displayed on the display screen 82. Note that in theexample shown in FIG. 9 , the composite image does not include auxiliaryinformation. An example of a composite image including auxiliaryinformation will be described later with reference to FIG. 19 .

4. Other Examples of Composite Images

Specific examples of composite images displayed on the display screen 82of the tablet terminal 80 according to the present embodiment aredescribed below with reference to FIGS. 16 to 21 . Note that in thefollowing examples, differences from the composite image 100 shown inFIG. 9 are mainly described, and a description of common points will beomitted or simplified.

4.1 Example of Composite Image which is a Still Image (for a Case wherethere is Only One Type of Object)

FIG. 16 shows another example of a composite image displayed on thedisplay screen 82 of the tablet terminal 80 according to the presentembodiment. In this specific example shown in FIG. 16 , a compositeimage 110 is displayed on the display screen 82.

The composite image 110 is an image obtained by combining a capturedimage 101 and aerosol images 112 and 114. The aerosol images 112 and 114are each an example of a second image representing at least one type ofaerosol existing in the space 95. In the example shown in FIG. 16 , theaerosol images 112 and 114 each represent pollen.

As shown in FIG. 16 , the aerosol image 112 includes a contour 112 a anddistance information 112 b. Similarly, the aerosol image 114 includes acontour 114 a and distance information 114 b.

In the example shown in FIG. 16 , the distance information 112 b is acolor determined depending on the distance and applied to the inside ofthe contour 112 a. In this regard, the distance information 114 b issimilar to the distance information 112 b. For example, a type ordarkness/lightness of a color is predetermined as a function of thedistance. Note that in FIG. 16 , the color is represented by the densityof dots inside the contour 112 a. For example, the color applied insidethe contour 114 a as the distance information 114 b is darker than thecolor applied inside the contour 112 a as the distance information 112b. Thus, the composite image 110 indicates that the pollen representedby the aerosol image 114 is shorter in distance than the pollenrepresented by the aerosol image 112.

Note that the distance information 112 b and 114 b may be represented bydensities of shades instead of colors. For example, the distance may berepresented by the density of dots inside the contour 112 a or 114 a.

The aerosol image 112 further includes level information 112 c, and theaerosol image 114 further includes level information 114 c. The levelinformation 112 c indicates a type and a density of aerosol representedby the aerosol image 112. The density is represented by the levelinformation 112 c by, for example, a representative value of controllevels Ci at respective coordinate points inside the contour 112 a. Morespecifically, for example, the level information 112 c indicates amaximum value or an average value of the control levels Ci at respectivecoordinate points inside the contour 112 a. In the example shown in FIG.16 , the level information 112 c includes characters representing pollenas a type of aerosol, and a numerical value representing the controllevel Ci. In this regard, the level information 114 c indicatesinformation in a similar manner.

As described above, in the composite image 110, the distance to theaerosol is displayed in a display mode other than a numerical value, andthus it is possible to prevent the image from including a large numberof characters indicating numerical values and thus prevent the imagefrom being complicated. The displaying of the distance in a display modeother than a numerical value allows it to use a numerical value and acharacter to represent the density of the aerosol. As a result, it ispossible to increase the amount of information presented to a user whilesuppressing complication in the image.

4.2 Example of Composite Image which is a Still Image (for a Case wherethere are Two or More Object)

Next, an example of a composite image is described for a case where twoor more types of aerosols exist in the space 95.

FIG. 17 shows an example of a composite image displayed on the displayscreen 82 of the tablet terminal 80 according to the present embodiment.That is, in FIG. 17 , a composite image 120 is displayed on the displayscreen 82.

The composite image 120 is an image in which a captured image 101 andaerosol images 122, 124, 126, and 128 are combined. The aerosol images122, 124, 126 and 128 each represent at least one type of aerosolexisting in the space 95. In the example shown in FIG. 17 , the aerosolimages 122 and 128 represent pollen, and the aerosol images 124 and 126represent dust.

As shown in FIG. 17 , the aerosol image 122 includes a contour 122 a anddistance information 122 b. The aerosol image 124 includes a contour 124a and distance information 124 b. The aerosol image 126 includes acontour 126 a and distance information 126 b. The aerosol image 128includes a contour 128 a and distance information 128 b. Each of thedistance information 122 b, 124 b, 126 b, and 128 b is a numerical valueindicating a distance, as in the composite image 100 shown in FIG. 9 .

In the composite image 120 shown in FIG. 17 , the aerosol image 122further includes level information 122 c. The aerosol image 124 furtherincludes level information 124 c. The aerosol image 126 further includeslevel information 126 c. The aerosol image 128 further includes levelinformation 128 c.

The level information 122 c is a color or a shade applied to the insideof the contour 122 a. More specifically, the level information 122 crepresents the magnitude of the control level Ci by darkness/lightnessof a color or a density of shades. For example, the level information122 c indicates the control level Ci such that the darker the color orthe denser the shade, the higher the control level Ci, while the lighterthe color or the sparser the shade, the lower the control level Ci. Thecontrol level Ci is represented in a similar manner also by the levelinformation 124 c, 126 c, and 128 c.

Furthermore, the level information 122 c represents a type of aerosol bya type of color or shade. That is, the same type of aerosol isrepresented by the same type of color or shade. For example, in theexample shown in FIG. 17 , the dot mesh represents pollen and the gridmesh represents dust. In the level information 124 c, 126 c, and 128 c,types of aerosol are represented in a similar manner.

As can be seen from the above description, the aerosol image 122represents the same type of aerosol as that represented by the aerosolimage 128. However, in the aerosol represented in the aerosol image 122,the density is low and the distance is far compared to those of theaerosol represented in the aerosol image 128. Similarly, the aerosolimage 124 represents the same type of aerosol as that represented by theaerosol image 126. However, in the aerosol represented in the aerosolimage 124, the density is high and the distance is close compared tothose of the aerosol represented in the aerosol image 126.

FIG. 18 shows another example of a composite image displayed on thedisplay screen 82 of the tablet terminal 80 according to the presentembodiment. As shown in FIG. 18 , a composite image 130 is displayed onthe display screen 82.

The composite image 130 is an image in which a captured image 101 andaerosol images 132, 134, 136, and 138 are combined. The aerosol images132, 134, 136 and 138 each represent at least one type of aerosolexisting in the space 95. In the example shown in FIG. 18 , the aerosolimages 132 and 138 represent pollen. The aerosol images 134 and 136represent dust.

As shown in FIG. 18 , the aerosol image 132 includes a contour 132 a,distance information 132 b, and level information 132 c. The aerosolimage 134 includes a contour 134 a, distance information 134 b, andlevel information 134 c. The aerosol image 136 includes a contour 136 a,distance information 136 b, and level information 136 c. The aerosolimage 138 includes a contour 138 a, distance information 138 b, andlevel information 138 c.

Each of the distance information 132 b, 134 b, 136 b, and 138 b is acolor predetermined depending on the distance and applied to the insideof a contour as with the case of the composite image 110 shown in FIG.16 . In the example shown in FIG. 18 , the distance information 132 b,134 b, 136 b, and 138 b each represent a type of aerosol by a type ofcolor or shade. That is, the same type of color or shade means the sameaerosol. In the example shown in FIG. 18 , a dot mesh represents pollenand a grid mesh represents dust.

Each of the level information 132 c, 134 c, 136 c, and 138 c includescharacters representing pollen as a type of aerosol, and a numericalvalue indicating a control level Ci, as in the composite image 110 shownin FIG. 16 .

As described above, types of aerosol are displayed in different displaymodes depending on the types, it is possible to present not only thepositions of the aerosol but also the types to a user. Thus, it ispossible to increase the amount of information presented to the userwhile suppressing complication in the image.

4.3 Example of Composite Image which is a Still Image (for a Case wherethe Composite Image Includes a Warning Image)

Next, an example of a composite image is described which is displayedwhen the density of aerosol exceeds a threshold value.

FIG. 19 shows an example of a composite image displayed on the displayscreen 82 of the tablet terminal 80 according to the present embodiment.As shown in FIG. 19 , a composite image 140 is displayed on the displayscreen 82.

The composite image 140 has an aerosol image 148 instead of the aerosolimage 138 included in composite image 130 shown in FIG. 18 . The aerosolimage 148 includes a contour 148 a, distance information 148 b, andlevel information 148 c. A contour 148 a, distance information 148 b,and level information 148 c are similar to the contour 138 a, thedistance information 138 b, and the level information 138 c shown inFIG. 18 .

The level information 148 c of the aerosol image 148 indicates that thecontrol level Ci is “3”. Since the control level Ci exceeds thethreshold value, a warning image 141 to provide a warning is displayedon the display screen 82.

The warning image 141 may be, by way of example but not limitation, acharacter that provides a warning. The warning image 141 may be, forexample, a particular graphical figure or the like. The mode ofdisplaying the warning image 141 is not particularly limited as long asit can attract a user's attention. For example, the entire compositeimage 140 displayed on the display screen 82 may be displayed in ablinking manner, or the color tone may be changed.

In addition to the warning image 141, or instead of the warning image141, preventive advice may be displayed on the display screen 82. Thepreventive advice may be displayed, for example, as characterinformation. Alternatively, instead of the character informationrepresenting the preventive advice, a URL (Uniform Resource Locator) ora QR code (registered trademark) for connecting to a web page in whichdetails of the preventive advice are described may be displayed.

4.4 Pseudo Three-Dimensional Image

Next, an example is described below fora a case where a composite imageis a pseudo three-dimensional image,

FIG. 20 shows an example of a composite image displayed on the displayscreen 82 of the tablet terminal 80 according to the present embodiment.As shown in FIG. 20 , a composite image 200 is displayed on the displayscreen 82.

The composite image 200 is a three-dimensionally modeled image in whichthe space 95 and a contour representing a boundary of a region in whichat least one type of aerosol exists are represented. More specifically,the composite image 200 is a pseudo three-dimensional image whoseviewpoint can be changed.

As shown in FIG. 20 , the composite image 200 is an image in which acaptured image 201 and an aerosol image 202 are combined. The aerosolimage 202 includes a contour 202 a and level information 202 c.

In part (a) of FIG. 20 , the composite image 200 is displayed on thedisplay screen 82 so as to show a view of the space 95 seen in ahorizontal direction, as in FIG. 9 . According to an instruction issuedby a user or as time passes, the composite image 200 is displayed on thedisplay screen 82 so as to show a view of the space seen obliquely fromabove, as shown in part (b) of FIG. 20 . A user is allowed to freelychange the viewpoint, for example, by swiping the display screen 82. Thecomposite image 200 may be displayed such that it is allowed to freelyenlarge or reduce the composite image.

When the viewpoint is changed, the position and the shape of the contour202 a of the aerosol image 202 are correspondingly changed. This allowsit to accurately present the position of the aerosol in the space 95.

4.3 Moving Image

Next, a composite image representing aerosol is described for a casewhere the composite image is a moving image.

FIG. 21 shows an example of a composite image displayed on the displayscreen 82 of the tablet terminal 80 according to the present embodiment.As time passes, the content of the composite image displayed on thedisplay screen 82 changes as illustrated in part (a) of FIG. 21 to part(e) of FIG. 21 . The content of the composite image 300 displayed on thedisplay screen 82 is sequentially switched at intervals of, for example,1 second to several seconds.

The composite image 300 is an image in which a captured image 301 andaerosol images 312, 322, 332, and 342 are combined. The aerosol images312, 322, 332 and 342 correspond to distance from the referenceposition.

As shown in FIG. 21 , distance information 302 is displayed on thedisplay screen 82. The distance information 302 represents the distancein the depth direction by a numerical value. The aerosol images 312,322, 332 and 342 represent aerosols at distances of 0.8 m, 1.1 m, 1.4 mand 1.7 m, respectively.

As shown in part (a) of FIG. 21 , no aerosol image is included in thecomposite image 300 when the distance is 0.5 m. That is, there is noaerosol at a distance of 0.5 m.

The aerosol image 312 includes a contour 312 a and level information 312c. The aerosol image 322 includes a contour 322 a and level information322 c. The aerosol image 332 includes a contour 332 a and levelinformation 332 c. The aerosol image 342 includes a contour 342 a andlevel information 342 c.

Contours 312 a, 322 a, 332 a, and 342 a respectively representboundaries of regions in which aerosol existing at correspondingdistances. Similarly, level information 312 c, 322 c, 332 c, and 342 crepresent the densities of the aerosol at the corresponding distances.More specifically, for example, the level information 312 c, 322 c, 332c and 342 c represents the maximum values of the densities ofcoordinates points inside the contours of the aerosol at thecorresponding distances. As shown in part (d) of FIG. 21 , the aerosolhas a highest control level Ci, that is, a highest density, when thedistance is 1.4 m.

In the composite image 300, the captured image 301 is a still image, butit may change with time. That is, the captured image 301 may be a movingimage.

OTHER EMBODIMENTS

The display apparatus, the image processing apparatus, and the controlmethod according to one or more aspects have been described above withreference to embodiments. However, the present disclosure is not limitedto those embodiments. It will be apparent to those skilled in the artthat many various modifications may be applicable to the embodimentswithout departing from the spirit and scope of the present disclosure.Furthermore, constituent elements of different embodiments may becombined. Any such resultant modifications also falls within the scopeof the present disclosure.

For example, in the above-described embodiments, it is assumed by way ofexample, the first sensor 30, the second sensor 40, and the third sensor50 are each an autonomous mobile sensor. However, the sensors are notlimited to those of the autonomous mobile type. At least one of thefirst sensor 30, the second sensor 40, and the third sensor 50 may be astationary sensor apparatus fixed at a particular position in the space95. The particular position is, for example, on a ceiling, a floor, or awall of the space 95.

For example, to reflect the density on an image representing an aerosolor another object, a numerical of the density may be displayed insteadof the control level. To indicate types of aerosol such that types aredisplayed in different modes depending on the types, line type ofcontours may be varied depending on the types of aerosol. For example,pollen may be represented by a solid contour line and dust may berepresented by a dashed contour line.

For example, the non-contact sensing system 10 may not include thecamera 20. A captured image may be obtained in advance by imaging thespace 95 and may be stored in the memory 66 of the computer 60.

For example, at least one of the first sensor 30, the second sensor 40,and the third sensor 50 may be a contact sensor.

The communication method between apparatuses in the above-describedembodiments is not particularly limited. When wireless communication isperformed between apparatuses, a wireless communication method (acommunication standard) may be, for example, ZigBee (registeredtrademark), Bluetooth (registered trademark), or a short-distancewireless communication such as wireless LAN (Local Area Network).Alternatively, the wireless communication method (the communicationstandard) may be that performed via a wide area communication networksuch as the Internet. Instead of wireless communication, wiredcommunication may be performed between apparatuses. More specifically,the wired communication may be a power line communication (PLC) or acommunication using a wired LAN.

In the above-described embodiments, processes executed by a specificprocessing unit may be executed by another processing unit. The order ofexecuting processes may be changed, or two or more processes may beexecuted in parallel. The above-described manner of allocating theconstituent elements of the non-contact sensing system 10 amongapparatuses is merely an example. For example, instead of allocating aconstituent element to a particular apparatus, the constituent elementmay be allocated to another apparatus. Conversely, the non-contactsensing system 10 may be realized in a single apparatus.

FIG. 22 is a diagram showing a tablet terminal 480 integrally providedwith the non-contact sensing system 10 according to an embodiment. Thetablet terminal 480 is a plate-shaped device. Part (a) of FIG. 22 andpart (b) of FIG. 22 are plan views showing one surface and the othersurface of the tablet terminal 480.

As shown in part (a) of FIG. 22 , a display screen 482 is provided on asurface of one side of the tablet terminal 480. As shown in part (b) ofFIG. 22 , a camera 20, a light source 32, and a photodetector 34 areprovided on a surface of the other side of the tablet terminal 480.Although not shown in FIG. 22 , the tablet terminal 480 includes aprocessor 64 and a memory 66 of a computer 60 according to anembodiment. As described above, in the tablet terminal 480, the displayscreen 482 that displays the composite image 481 may be integrated withthe camera 20, the sensor apparatus, and the computer 60.

For example, a process performed by the server apparatus 70 may beperformed by the computer 60 or the tablet terminal 80. A processperformed by the computer 60 may be performed by the server apparatus 70or the tablet terminal 80.

In the above-described embodiments, it is assumed by way of example thatthe computer 60 generates composite images. However, the controller 84of the tablet terminal 80 may generate a composite image. Morespecifically, the controller 84 may perform the process, in FIG. 11 , ofconverting the captured image data into the 3D database and the processof converting the sensor data into the 3D database. The controller 84may perform, in FIG. 10 , the conversion into 3D database (S26), thegeneration of the level distribution (328), and the generation of thecomposite image (S32).

One or more processes described in the above embodiments may be realizedby performing centralized processing using a single apparatus or asystem, or may be realized by performing distributed processing usingtwo or more apparatuses. One or more programs may be executed by asingle processor or two or more processors. That is, processes may beperformed in the centralized manner or distributed manner.

In the above-described embodiments, all or part of the constituentelements such as the controller may be implemented by hardware or may berealized by executing software programs corresponding to the constituentelements. Each constituent element may be realized by reading a softwareprogram stored in a storage medium such as an HDD (Hard Disk Drive), asemiconductor memory, or the like and executing the software program bya program execution unit such as a CPU (Central Processing Unit), aprocessor, or the like.

Furthermore, the constituent elements such as the controller may beconfigured by one or more electronic circuits. The one or moreelectronic circuits each may be a general-purpose circuit or a dedicatedcircuit.

The one or more electronic circuits may include, for example, asemiconductor device, an IC (Integrated Circuit), or an LSI (Large ScaleIntegration). The IC or LSI may be integrated on one chip or may beintegrated on two or more chips. Note that the ICs or LSIs are calleddifferently, depending on the integration density, such as a system LSI,a VLSI (Very Large Scale Integration), or a ULSI (Ultra Large ScaleIntegration). Furthermore, an FPGA (Field Programmable Gate Array)capable of being programmed after the LSI is produced, thereby properlymay also be used for the same purpose.

General or specific embodiments may be implemented as a system, anapparatus, a method, an integrated circuit, or a computer program, orgeneral or specific embodiments may be implemented by acomputer-readable non-transitory storage medium such as an optical disk,an HDD, a semiconductor memory, or the like in which the computerprogram is stored. General or specific embodiments may be implemented byany selective combination of a system, an apparatus, a method, anintegrated circuit, a computer program, and a storage medium.

Also note that in each embodiment described above, various changes,replacements, additions, removals, or the Ike are possible withoutdeparting from scope of the present disclosure or equivalent scope.

The present disclosure may be used as a display apparatus or the like toaccurately present a precise position of aerosol, and may be used, forexample, for air conditioning control or space purification processingcontrol.

What is claimed is:
 1. A display apparatus comprising: a display screen; and a controller that causes the display screen to display a composite image in which a first image acquired by imaging a space by a camera and a second image representing at least one type of aerosol existing for real in the space of the first image are combined, wherein the at least one type of aerosol is a particle floating in the space, wherein a position of the at least one type of aerosol as seen in a depth direction in the first image is reflected in the second image, and wherein a density of the at least one type of aerosol is further reflected in the second image, the density of the at least one type of aerosol is obtained by irradiating an incident light to the at least one type of aerosol, and an intensity of a return light from the at least one type of aerosol corresponds to the density of the at least one type of aerosol.
 2. The display apparatus according to claim 1, wherein the first image represents a two-dimensional space, the controller generates the second image by projecting three-dimensional coordinate data representing the position of the at least one type of aerosol in the space onto the two-dimensional space, and the controller generates the composite image by combining the first image and the second image.
 3. The display apparatus according to claim 2, wherein the controller acquires the three-dimensional coordinate data from a sensor that acquires a position of the at least one type of aerosol in the space, the controller converts the first image into a pseudo three-dimensional image, and the controller generates the second image by projecting the three-dimensional coordinate data into the two-dimensional space such that the pseudo three-dimensional image and the three-dimensional coordinate data correspond to each other.
 4. The display apparatus according to claim 1, wherein the second image includes a contour representing a boundary of a region in which the at least one type of aerosol exists and distance information representing a distance in the space from a reference position to a representative position of the region inside the contour.
 5. The display apparatus according to claim 4, wherein the representative position is a center of gravity of a density distribution of the at least one type of aerosol in the region inside the contour.
 6. The display apparatus according to claim 4, wherein the distance information is a numerical value indicating the distance.
 7. The display apparatus according to claim 4, wherein the distance information is a color that is predetermined according to the distance and is applied to the region inside the contour.
 8. The display apparatus according to claim 1, wherein the composite image represents a three-dimensional model including the space and a contour representing a boundary of a region in which the at least one type of aerosol exists.
 9. The display apparatus according to claim 1, wherein the second image is a moving image including images that are switched as time passes, and each of the images corresponds to a distance from a reference position in the space, and includes a contour indicating a boundary of a region, at the corresponding distance, in which the at least one type of aerosol exists.
 10. The display apparatus according to claim 1, wherein the second image includes level information indicating a density level of the at least one type of aerosol.
 11. The display apparatus according to claim 1, wherein the at least one type of aerosol includes two or more types of aerosol, and the second image represents the respective two or more types of aerosol in different display modes.
 12. The display apparatus according to claim 1, wherein the controller further causes the display screen to display an image for warning a user in a case where the density of the at least one type of aerosol is greater than a threshold value.
 13. An image processing apparatus comprising: an acquisition circuit that acquires three-dimensional coordinate data representing a position, in a space, of at least one type of aerosol existing for real in the space; and a processor, wherein wherein the at least one type of aerosol is a particle floating in the space, the processor generates a composite image in which a first image acquired by imaging the space by a camera and a second image representing the at least one type of aerosol existing for real in the space of the first image are combined based on the three-dimensional coordinate data, a position of the at least one type of aerosol as seen in a depth direction in the first image is reflected in the second image, and a density of the at least one type of aerosol is further reflected in the second image, the density of the at least one type of aerosol is obtained by irradiating an incident light to the at least one type of aerosol, and an intensity of a return light from the at least one type of aerosol corresponds to the density of the at least one type of aerosol.
 14. A control method of controlling a system, the system comprising a sensor that includes a light source emitting irradiation light toward at least one type of object in a space and a photodetector detecting return light returning from the at least one type of object, the sensor outputting data representing a result of detection of the return light by the photodetector, and a display apparatus, the control method comprising: acquiring the data from the sensor: generating three-dimensional coordinate data representing a position, in the space, of the at least one type of object based on the data; based on the three-dimensional coordinate data, generating a composite image in which a first image and a second image are combined, the first image being obtained by imaging the space by a camera, the second image representing the at least one type of object existing for real in the space of the first image, reflecting a position of the at least one type of object as seen in a depth direction in the first image, wherein the at least one type of object is a particle floating in the space, and reflecting a density of the at least one type of object in the second image, wherein the density of the at least one type of object is obtained by irradiating an incident light to the at least one type of object, and an intensity of a return light from the at least one type of object corresponds to the density of the at least one type of object; and causing the display apparatus to display the composite image.
 15. The control method according to claim 14, wherein the return light is fluorescent light emitted by the at least one type of object by being excited by the irradiation light, and the generating of the composite image includes determining a type of the at least one type of object by analyzing the fluorescent light and reflecting the type in the second image.
 16. The control method according to claim 15, wherein the irradiation light includes a polarization component, and the generating of the composite image includes determining the type of the at least one type of object based on degree of depolarization of the polarization component included in the return light and reflecting the type in the second image.
 17. The control method according to claim 14, wherein the three-dimensional coordinate data is generated using coordinates of a position of the sensor in the space and a relative positional relationship between the sensor and the at least one type of object calculated based on a difference between an irradiation light emission time and a return light reception time.
 18. The control method according to claim 14, wherein the at least one type of object is an organic substance stuck to an object existing in the space.
 19. The control method according to claim 14, wherein the at least one type of object is aerosol existing in the space.
 20. The control method according to claim 19, wherein the return light is backscattered light generated as a result of scattering of the irradiation light by the at least one type of object.
 21. A non-transitory computer-readable storage medium storing a program for controlling a system, the system comprising a sensor that includes a light source emitting irradiation light toward at least one type of object in a space and a photodetector detecting return light returning from the at least one type of object, the sensor outputting data representing a result of detection of the return light by the photodetector, and a display apparatus, the program, when executed by the computer, performing: acquiring the data from the sensor; generating three-dimensional coordinate data representing a position, in the space, of the at least one type of object based on the data; based on the three-dimensional coordinate data, generating a composite image in which a first image and a second image are combined, the first image being obtained by imaging the space by a camera, the second image representing the at least one type of object existing for real in the space of the first image, reflecting a position of the at least one type of object as seen in a depth direction in the first image, wherein the at least one type of object is a particle floating in the space, and reflecting a density of the at least one type of object in the second image, wherein the density of the at least one type of object is obtained by irradiating an incident light to the at least one type of object, and an intensity of a return light from the at least one type of object corresponds to the density of the at least one type of object, and causing the display apparatus to display the composite image. 